The pitfalls of using open data to develop deep learning solutions for COVID-19 detection in chest X-rays (preprint)

Since the emergence of COVID-19, deep learning models have been developed to identify COVID-19 from chest X-rays. With little to no direct access to hospital data, the AI community relies heavily on public data comprising numerous data sources. Model performance results have been exceptional when training and testing on open-source data, surpassing the reported capabilities of AI in pneumonia-detection prior to the COVID-19 outbreak. In this study impactful models are trained on a widely used open-source data and tested on an external test set and a hospital dataset, for the task of classifying chest X-rays into one of three classes: COVID-19, non-COVID pneumonia and no-pneumonia. Classification performance of the models investigated is evaluated through ROC curves, confusion matrices and standard classification metrics. Explainability modules are implemented to explore the image features most important to classification. Data analysis and model evaluations show that the popular open-source dataset COVIDx is not representative of the real clinical problem and that results from testing on this are inflated. Dependence on open-source data can leave models vulnerable to bias and confounding variables, requiring careful analysis to develop clinically useful/viable AI tools for COVID-19 detection in chest X-rays.

Development and validation of multivariable machine learning algorithms to predict risk of cancer in symptomatic patients referred urgently from primary care (preprint)

Background Urgent Suspected Cancer (Two Week Wait, 2WW) referrals have improved early cancer detection but are increasingly a major burden on NHS services. This has been exacerbated by the COVID-19 pandemic. Method We developed and validated tests to assess the risk of any cancer for 2WW patients. The tests use routine blood measurements (FBC, U&E, LFTs, tumour markers), combining them using machine learning and statistical modelling. Algorithms were developed and validated for nine 2WW pathways using retrospective data from 371,799 referrals to Leeds Teaching Hospitals Trust (development set 224,669 referrals, validation set 147,130 referrals). A minimum set of blood measurements were required for inclusion, and missing data were modelled internally by the algorithms. Results We present results for two clinical use-cases. In use-case 1, the algorithms identify 20% of patients who do not have cancer and may not need an urgent 2WW referral. In use-case 2, they identify 90% of cancer cases with a high probability of cancer that could be prioritised for review. Conclusions Combining a panel of widely available blood markers produces effective blood tests for cancer for NHS 2WW patients. The tests are affordable, can be deployed rapidly to any NHS pathology laboratory with no additional hardware requirements.

Estimating excess mortality in people with cancer and multimorbidity in the COVID-19 emergency (preprint)

Background: Cancer and multiple non-cancer conditions are considered by the Centers for Disease Control and Prevention (CDC) as high risk conditions in the COVID-19 emergency. Professional societies have recommended changes in cancer service provision to minimize COVID-19 risks to cancer patients and health care workers. However, we do not know the extent to which cancer patients, in whom multi-morbidity is common, may be at higher overall risk of mortality as a net result of multiple factors including COVID-19 infection, changes in health services, and socioeconomic factors. Methods: We report multi-center, weekly cancer diagnostic referrals and chemotherapy treatments until April 2020 in England and Northern Ireland. We analyzed population-based health records from 3,862,012 adults in England to estimate 1-year mortality in 24 cancer sites and 15 non-cancer comorbidity clusters (40 conditions) recognized by CDC as high-risk. We estimated overall (direct and indirect) effects of COVID-19 emergency on mortality under different Relative Impact of the Emergency (RIE) and different Proportions of the population Affected by the Emergency (PAE). We applied the same model to the US, using Surveillance, Epidemiology, and End Results (SEER) program data. Results: Weekly data until April 2020 demonstrate significant falls in admissions for chemotherapy (45-66% reduction) and urgent referrals for early cancer diagnosis (70-89% reduction), compared to pre-emergency levels. Under conservative assumptions of the emergency affecting only people with newly diagnosed cancer (incident cases) at COVID-19 PAE of 40%, and an RIE of 1.5, the model estimated 6,270 excess deaths at 1 year in England and 33,890 excess deaths in the US. In England, the proportion of patients with incident cancer with [≥]1 comorbidity was 65.2%. The number of comorbidities was strongly associated with cancer mortality risk. Across a range of model assumptions, and across incident and prevalent cancer cases, 78% of excess deaths occur in cancer patients with [≥]1 comorbidity. Conclusion: We provide the first estimates of potential excess mortality among people with cancer and multimorbidity due to the COVID-19 emergency and demonstrate dramatic changes in cancer services. To better inform prioritization of cancer care and guide policy change, there is an urgent need for weekly data on cause-specific excess mortality, cancer diagnosis and treatment provision and better intelligence on the use of effective treatments for comorbidities.